texts:
I'd like to understand when using PG methods is more adequate than using DQN methods. Just to give a bit of background: I am currently using both APEX and R2D2 for my projects. Both work very well in achieving convergence across a variety of tasks, which I cannot say for PG based methods, such as PPO. One of the major differences of course being off-policy vs on-policy. Basically I think that having a replay buffer is incredibly useful, especially for tasks/games that do not have continuous rewards to guide the algorithm, such as MountainCar. Certain PG algorithms use off-policy correcting mechanisms, such as IMPALA using V-trace, etc, which helps, but is not truly allowing for off-policy behavior.

So to be more specific: If you don't require continuous action space, and you compare APEX vs PPO - at what point does it make more sense to use PPO? Are there any clear examples/situations where PPO would significantly outperform APEX? Anything where PPO would converge and APEX wouldn't?


texts: How is data science applied in investment banking?

texts: What do data scientists do at investment banks? What tools are they using? What kind of analysis are they doing? Why are they doing it? Etc.


texts: Loss function to maximize sum of targets

texts:
I have a dataset {$X_i$, $Y_i$}. $Y_i$ - real value targets, they can be negative. The task is to teach classifier $f: X_i \rightarrow \{0, 1\}$ to maximize the sum below
$$
L(f) = \sum_i f(X_i) * Y_i
$$
So $f(X_i) = I[Y_i > 0]$ would be the ideal classifier.
I'm looking for appropriate smooth loss function to teach NN. NN output is probability $p(X_i)$, so $f(X_i) = I[p(X_i) > 0.5]$. I have tried the next loss function:
$$
BadLoss(p) = \sum_i p(X_i) * Y_i
$$
but because $Y_i$ can be negative it is bad-defined loss and NN converges to nothing good. The best results I got were with usual logloss:
$$
Loss(p) = \sum_i logloss(p(X_i), \; I[Y_i > 0])
$$

Can be there better loss functions to maximize the sum $L(f)$?


texts: On Creating an Interoperable Network Matrix

texts:
Complete self taught novice here.
Question
Are there any popular standards, or commonly used formats for representing/saving/sharing/serializing a network?
Background
I built a simple but very large (5.5m nodes, 16.7m edges) network matrix using my own python code. But if I wanted to use this matrix with other data science tools what format should I use?
Currently I have it in a giant csv file with headers and "1" representing edges. Which I think is called a binary matrix.
Goal
I want to make use of any benefits present in using binary files, or formatted flat files that would help me learn other tools.


texts: Getting wrong ch2 values from sklearn chi2

texts:
I have a feature vector table which looks like this-

This a table with 156 columns or features.I want to apply feature selection algorithm to this before applyinh my classification model.

This is what I am using-

dataset = pd.read_csv('.csv')
X = dataset.iloc[:, 1:157].values
y = dataset.iloc[:,0].values

##normalize

scaler = MinMaxScaler()
scaler.fit(X)
MinMaxScaler(copy=True, feature_range=(0, 1))

X_normalized = scaler.transform(X)

##feature selection

sel = SelectKBest(chi2, k='all')

sel.fit_transform(X_normalized, y)

print(sel.scores_)

this is the result of print(sel.scores_) I am getting-

As can be seen they are not all between 0 and 1.

I a referring this research paper as my source-

source-http://courses.ischool.berkeley.edu/i256/f06/papers/yang97comparative.pdf


texts:
I have a data set like the following

where the rows are dates and the columns are values recorded by different sensors on those dates.
Before working with the data for the purpose of predicting it, I would like to remove the anomalies.
Standard deviation
The first thing I found on the internet was to get data out of the standard deviation
so working with this, I've come up with things like this.

I search a little more in interet I have read that it is not a good method when the data does not follow a normal distribution.
So I did a shapiro test for normality and the H0 hypothesis was rejected, my data doesn't look like a Guassian
But if I plot a histogram, the data looks like half a bell.

IQR in some sites I read about the IQR method which is based on ordering the data, so I went to test it too, and with a value of n of 1.5 it gave me results equal to the standard deviation

tsmoothie At this point I was quite confused and I found this library and this link
I thought maybe it was a better way than the above and tried to test it too.
But I ran into the problem that it has many methods and I honestly don't know which one to use.
PCA + IQR I kept looking on the internet and I found this link that left me more confused, it is a case similar to mine where there are several time series of various sensors.
But after applying pca and iqr the following are anomalous points

For me, this sensor has a normal behavior, but it marks many anomalous points, and in sensors with points  with high values, it does not mark them as anomalous
So in this context, what would be the best approach to detect the anomalous points before working with the data? What do you get used to doing?


texts: Problem of multi class classification (Sklearn TfidfVectorizer and SGDClassifier)

texts:
I do the (text) topic classification using TfidfVectorizer and SGDClassifier, literally I want to classify the website into categories (like Sport, Business etc).
Now, the problem is, that each website might fit into multiple categories eg (IT and Eshop, Sport and Eshop) etc.
The question can have 2 parts on is how to do it technically (Python, SKLEARN) another is a theoretical question.
Let me explain the latter.
As far as I understand how the text classification works, it cannot work for multiclass (eg adding another Eshop class output) because, the IT eshop is full of IT keywords => it is easy to classify it as IT related site.
But if i take 3 IT eshops sites and add it in the testing data both as IT and Eshop category, the keywords (word vectors) for these both categories will be same, and literally only IT focused.
Let's say I add 3 more Sport eshops and the problem is still same. The new category Eshop won't get only the clean expected keywords like "Shopping cart, Checkout" etc, but it will keep the same word vectors for IT and Sport.
Am I right, yes?
Even if I had some extra solo classification to mark the eshops or some meta categories like Eshop, Small business, the problem is still same, it will still keep the top words which are more or less related to topic like IT, Sport.
So the question is, how to do the multi class classification (I am ok with the 1 class output of SGDClassifier as long as it will mark more classes with .predict_proba() method)
Then how to deal with this problem of multi classes like IT + Eshop, or IT + something else.
And then I realize that Eshop is maybe too much "meta" category so maybe it needs some other logic or idea, than TfidfVectorizer. (Maybe some category related keywords across all documents in same category as I mentioned above, or other method like Neural Network or I don't know).


texts: Evaluate Model Node in Azure ML Studio does not take all the rows of the dataset in confusion matrix

texts:
I have this dataset in which the positive class consists of component failures for a specific component of the APS system.
I am doing Predictive Maintenance using Microsoft Azure Machine Learning Studio.
As you can see from the pictures below, I am using 4 algorithm: Logistic Regression, Random Forest, Decision Tree and SVM. And you can see that the Output dataset in the score model node consists of 16k rows. However, when I see the output of the Evaluate Model, in the confusion matrix there are only 160 observations for the Logistic Regression, and the correct number, 16k for Random Forest. I have the same problem, only 160 observations in the models of Decision Tree and SVM. And the same problem is repeated in other experiments for example after feature selection, normalization etc.: some evaluate model does not use all the rows of the test dataset, and some other node does it.
How can I fix this problem? Because I am interested in the real number of false positive and false negatives.


texts: Detect sensitive data from unstructured text documents

texts:
I know this question is broad, but I need an advice to know if it's possible to achieve what I want to do.

The problem is that I have around 2500 documents with sensitive data being replaced by four dots. I do not have the original documents, so I wonder if there is a way to build a model that can detect sensitive data from any new documents (with sensitive data not being removed) using the previous documents? I want to apply machine learning or deep learning approaches. And what I know is that the original data set with annotated sensitive data should be used for training, which I can't obtain.

I am new at this field so any advice would be very appropriated


texts: Cross Validation how to determine when to Early Stop?

texts:
When using "K-Fold Cross Validtion" for Neural Net, do we:

Pick and save initial weights of the network randomly (let's call it $W_0$)
Split data into $N$ equal chunks
Train model on $N-1$ chunks, validating against the left-out chunk (the $K$'th chunk)
Get validation error and revert the weights back to $W_0$
shift $K$ by 1 and repeat from 3.
Average-out the validtion errors, to get a much better understanding of how network will generalize using this data.
Revert back to $W_0$ one last time, and train the network using the ENTIRE dataset

Question 1:

I realize 7 is possible, because we have a very good understanding of how network will generalize with the help of step 6. - Is this assumption correct?

Question 2:

Is reverting back to the initial $W_0$ a necessity, else we would overfit? (revert like we do in step 4. and 7.)

Question 3, most important:

Assume we've made it to step 7, and will train the model using ENTIRE data. By now, we don't intend to validate it after we will finish. In that case how do we know when to stop training the model during step 7?

Sure, we can train with same number of epochs as we did during Cross validation. But then how can we be sure that Cross Validation was trained with an appropriate number of epochs in the first place?

Please notice - during steps 3, 4, 5 we only had $K$'th chunk to evaluate Training vs Validation loss. $K$'th chunk is very small, so during the actual Cross-Validation it was unclear when to Early-Stop... To make things worse, it will be even more difficult in case of Leave-One-Out (also know as All-But-One), where K is simply made from a single training example


texts: What program to use to visualise neural network diagram and math functions

texts: I am writing a paper about machine learning and I need to create some neural network diagrams and basic math functions I am describing. I need a program to create visually decent technical picture without spending days to learn how to use the program.


texts: Should I fit my parameters with brute force

texts:
I am running analysis on data for this type of sensor my company makes. I want to quantify the health of the sensor based on three features using the following formula:

sensor health index = feature1 * A + feature2 * B + feature3 *C

We also need to pick a threshold so that if this index exceeds the threshold, the sensor is considered as bad sensor.

We only have a legacy list which shows about 100 sensors are bad. But now we have data for more than 10,000 sensors. Anything not in that 100 sensor list is NOT necessarily bad. So I guess the linear regression methods don't work in this scenario.

The only way I can think of is the brute force fitting. Pseudo code is as follows:

# class definition for params(coefficients)
class params{
  a
  b
  c
  th
}

# dictionary of parameter and accuracy rate
map = {}

for thold in range (1..20):
   for a in range (1..10):
      for b in range (1..10):
        for b in range (1..10):
           # bad sensor list
           bad_list = []
           params = new params[a, b, c, thold]
           for each sensor:
             health_index = sensor.feature1*a+sensor.feature2*b+sensor.feature3*c
             if health_index > thold:
               bad_list.append(sensor.id)
           accuracy = percentage of common sensors between bad_list and known_bad_sensors
           map[params] = accuracy

# rank params based on accuracy
rank(map)
# the params with most accuracy is the best model
print map.index(0)

I really don't like this method since it is using 5 for loops which is very efficient. I wonder if there is a better way to do it. Using existing library such as sk-learn perhaps?


texts:
As we all know there are two types of seasonal types, additive and multiplicative, but I have trouble telling them apart. To my understanding, in multiplicative seasonality, the magnitude of a seasonality would grow with the trend. For example with an upward trend, the magnitude of a season would grow bigger. But what if the magnitude of a seasonality grows but the trend stays the same. In other words, the minimum value of one season always seem to be on one horizontal line but the maximum value keeps increasing. What kind of seasonality is this? What model can I use to describe this data series?

Edit:

This is the formula given in Forecasting, Principles and Practice. From what I read it seems that the trend and the level are not multiplied together in the multiplicative seasonality case. Also in the statsmodels library, both the trend and the seasonality can be chosen as either multiplicative or additive. Can anyone clarify more what kind of additive mode we should be talking about here?


texts:
Will images modification get me a better machine learning model?
I have the following scenario. Camera is fixed and does photos of a process. The process has a few states. Now I want to train a model given a photo to classify to which state, does the photo belong. I.e. what is the current state of the process.
For the model training I was given only one photo for each of the process state.
Now my question is whether I should train my model with one photo in each class or should I take that one photo and do random modifications to the photo so that to get a few photos for each class?
I am thinking that it is better for the training to have just one but real photo. Instead of a lot modified. Am I right?
I use the ML.NET plugin for a Visual Studio to train the image classification model.


texts: Classification problem using features with unequal sizes

texts:
I am relatively new to Machine Learning/ Deep Learning and currently I am working on a classification problem. I have many 2D images and each of them is a cross section of a specimen showing the deformation after applying a load. The images show the edges of the specimen with the same color and the deformed edges are shown with different colors. I want to use the values of pixels as one of the feature to my classification problem. At the same time I also want to use the load value, the co-ordinates of the position of impact as features. Therefore my problem would look something like this

     Feature 1           Feature 2          Feature 3         Class
Vector of size N*1  Vector of size 3*1   Scalar value m1     Class A
Vector of size N*1  Vector of size 3*1   Scalar value m2     Class B

and so on..

I would like to know if there is a way to deal with this kind of problem or if there are any papers which focus on approaches to deal something similar to this. Any kind of help is really appreciated. If any clarification is needed from my side, do let me know.
Thank you.


texts: Spatial clustering of data points on a grid to obtain variable resolution map with constant statistical confidence

texts:
I have a grey valued image which is calculated as the mean of a series of images. The value of each pixel is therefore associated to a standard error.

The pixel values and the relative standard error (RSE) vary smoothly in pixel space, so that the pixels of a downscaled image resulting as the mean of the underlying downscaled images would exhibit a lower RSE. Meaning that a high resolution image is associated to high RSEs and a low resolution image to lower RSEs. However, the RSE vary in space.

Effectively this behaves like the trade-off of an uncertainty relation: Resolution in space vs resolution in the mean value.

For representing this I am looking for a way to divide the image into regions with constant RSEs, which would be larger in parts of the image with high RSEs and smaller in parts with low RSEs.

The aim would be to obtain irregularly sized superpixels with regular RSEs as opposed to my current image which contains pixels on a regular grid but with irregular RSEs.

This would be equivalent to downscaling the image but not on a regular grid. One approach I tried is agglomerative clustering of the pixels computing new distances in every step, but so far I couldn't find the right distance and objective functions to obtain a more or less regular map of superpixels (generally one big cluster would eat up its neighbours).

Do you know of any existing approach for this or if not which would be a good clustering method to apply? Would quadtrees help me?

Thanks a lot in advance!


texts:
I am having a hard time trying to understand the MSE loss function given in the Introduction to Boosted Trees (beware! My maths skills are the equivalent of a very sparse matrix):

$
\begin{split}\text{obj}^{(t)} & = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\
          & = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + constant
\end{split}
$

The second equality sign implies that one could easily derive the second equation from the first one, but I cannot see how.
My first naïve attempt was to:

express $y_i$ as $a$
express $(\hat{y}_i^{(t-1)} + f_t(x_i))$ as $b$
and then expand $(a-b)^2$

But I wasn't successful. Any help is really appreciated.


texts: shifting specific column to before/after specific column in dataframe

texts:
In dataframe example :

  medcine_preg_oth medcine_preg_oth1 medcine_preg_oth2 medcine_preg_oth3
0          Berplex           Berplex              None              None
1              NaN               NaN               NaN               NaN
2              NaN               NaN               NaN               NaN
3            obmin             obmin              None              None
4              NaN               NaN               NaN               NaN

'medcine_preg_oth1' 'medcine_preg_oth2' 'medcine_preg_oth3' ,these three columns are in somewhere of dataframe with other columns.

I want to shift these three : medcine_preg_oth1 medcine_preg_oth2 medcine_preg_oth3 to the place of after 'medcine_preg_oth'.

My idea is shifting the specific columns to place after/ before specific columns in dataframe for wider purpose .
please suggest me!
Thanks


texts: Text Mining - Similarity among words to determine thresholds

texts:
While trying to find how one place (Military High School) like a school is similar to another one (Military H School or Military High S), I used the LV algorithm to find the string distance and then converted them into a percentage on a calibrated scale.

stringdist(data[i,1],data[i,2],method = "lv")

Though this is admissible, I still need a more precise way to determine the threshold. The goal is to set a threshold of acceptance for a variation of a name from the standardized name, so that say if the threshold of this word is less than say 85%, it would be rejected. This exercise is part of data pre-processing.

Any suggestions.


texts: Many people use the mean and variance of the training set to standardize the test set, instead of calculating the mean and variance of the test set and use these. Isn't risky to do that ? If no, why ?


texts: How to test dev set on Time Series data via forecasting

texts:
I'm implementing $3$ Bayesian Deep Learning models (links below) for my masters. I'm supposed to test them on a civil engineering time series data.

My models should take a time series covariate vector ($X_t$ = {$x1_t$, $x2_t$, ...}) as input and predict single values $y_t$. I will use past values of $X$, (e.g. $X_{t-1}$, $X_{t-2}$) on each $y_t$ but the models won't be feed with past values of $y_t$ because these will not be available on a real situation. The data was generated daily, so every t is a new day. Summing up, the model shall take windows of X values (e.g. $X_{t-1}$, $X_{t-2}$ ...) with a fixed window size, and them predicting a single y value. We will have multiple pairs $((X_{t},X_{t-1}),y_t)$ as our training data, for multiple values of $t$.

I have several years of data, and I've separated the data on some arbitrary point T so that the first years are used as training, and the last months of the last year of data are used as test/dev data.

So, training examples are generated until $T (X_T,y_T)$, and them $T+1, T+2...$ are used to test the models.

The only value that matters on this forecasting task is the next day value ($y_{T+1}$).

Initially, I was training the model until day T, and then using that model to predict the annotations $y_{T+n}$ until the dev data stops. But then the people that gave me the data asked about retraining the model every new day of the dev set, and then predicting the next day:

For example, on the first training, I would train until day T and then forecast T+1. Then I would add the real value of $y_{T+1}$ to the train set, retrain the models, and forecast T+2 and so on.

Which approach would be more proper? The accuracy of the models drop when I try to retrain the models every day to predict just a single day, I can't understand why.

I'm using DeepAR, DeepFactors, Enc-Dec-Forecaster (by Uber) and an ARIMA model just to compare. I'd really appreciate some help!

EDIT: For clarity


texts:
can somebody recommend me any good resources on learning Elasticsearch for Data Science? I'm not interested in basics (how to query ES) but rather on books/tutorials on more advanced topics related to NLP or machine learning within ES in general.
Thanks


texts: High model accuracy vs very low validation accuarcy

texts:
I'm building a sentiment analysis program in python using Keras Sequential model for deep learning

my data is 20,000 tweets:

positive tweets: 9152 tweets
negative tweets: 10849 tweets

I wrote a sequential model script to make the binary classification as follows:

model=Sequential()
model.add(Embedding(vocab_size, 100, input_length=max_words))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(250, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model

print(model.summary())
history=model.fit(X_train[train], y1[train], validation_split=0.30,epochs=2, batch_size=128,verbose=2)

however I get very strange results! The model accuracy is almost perfect (>90)
whereas the validation accuracy is very low (<1) (shown bellow)

Train on 9417 samples, validate on 4036 samples
 Epoch 1/2
- 13s - loss: 0.5478 - acc: 0.7133 - val_loss: 3.6157 - val_acc: 0.0243
 Epoch 2/2
- 11s - loss: 0.2287 - acc: 0.8995 - val_loss: 5.4746 - val_acc: 0.0339

I tried to increase the number of epoch, and it only increases the model accuracy and lowers the validation accuracy

Any advice on how to overcome this issue?

Update:

this is how I handle my data

#read training data
pos_file=open('pos2.txt', 'r', encoding="Latin-1")
 neg_file=open('neg3.txt', 'r', encoding="Latin-1")
# Load data from files
pos = list(pos_file.readlines())
neg = list(neg_file.readlines())
x = pos + neg
docs = numpy.array(x)
#read Testing Data
pos_test=open('posTest2.txt', 'r',encoding="Latin-1")
posT = list(pos_test.readlines())
neg_test=open('negTest2.txt', 'r',encoding="Latin-1")
negT = list(neg_test.readlines())
xTest = posT + negT
total2 = numpy.array(xTest)

CombinedDocs=numpy.append(total2,docs)

# Generate labels
positive_labels = [1 for _ in pos]
negative_labels = [0 for _ in neg]
labels = numpy.concatenate([positive_labels, negative_labels], 0)

# prepare tokenizer
t = Tokenizer()
t.fit_on_texts(CombinedDocs)
vocab_size = len(t.word_index) + 1
# integer encode the documents
encoded_docs = t.texts_to_sequences(docs)
#print(encoded_docs)

# pad documents to a max length of 140 words
max_length = 140
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

Here I used Google public word2vec

# load the whole embedding into memory
embeddings_index = dict()
f = open('Google28.bin',encoding="latin-1")
for line in f:
values = line.split()
word = values[0]
coefs = asarray(values[1:], dtype='str')
embeddings_index[word] = coefs
f.close()

# create a weight matrix for words in training docs
embedding_matrix = zeros((vocab_size, 100))

for word, i in t.word_index.items():
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
    embedding_matrix[i] = embedding_vector

#Convert to numpy
NewTraining=numpy.array(padded_docs)
NewLabels=numpy.array(labels)
encoded_docs2 = t.texts_to_sequences(total2)

# pad documents to a max length of 140 words

padded_docs2 = pad_sequences(encoded_docs2, maxlen=max_length, padding='post')

# Generate labels
positive_labels2 = [1 for _ in posT]
negative_labels2 = [0 for _ in negT]
yTest = numpy.concatenate([positive_labels2, negative_labels2], 0)
NewTesting=numpy.array(padded_docs2)
NewLabelsTsting=numpy.array(yTest)


texts: Need some info regarding string matching algorithms?

texts:
Let me explain a scenario to better explain my question,

Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought fruits or not. Let's assume we used OCR to extract only the names of items bought and stored in a list.

First thing I did is web scrape all the names of fruits found everywhere and I stored the names of each of them in a big text file.

Now I want to know how can I match/lookup and make a decision that person bought fruits.

1) Any search/match algorithms that works on huge amounts of data.

I'm just looking for advice on what to implement in this type of scenario. Thanks in advance.


texts: Feature extraction in ConvNNs - layers progression

texts:
This is a bit of a confusion I have and I never saw any explanation of it, so I have to ask the question. Whenever I see the discussion on how a ConvNNs extracts features, I see something like this:

And the discussion that follows is that features are increasingly (hierarchically) more complicated and meaningful, as becomes obvious from the example here. Then, I extracted some layers from a ConvNN (CRF as RNN).

There are 64 feature maps in conv1_1, 256 in conv3_1 and 512 in conv5_1.
So my first problem is that I can't relate the results I have to the example above. I fail to see how the features become more complicated. All I see is that the image becomes increasingly less distinct. The final output (pixelwise prediction) is very good, so it must work. But I totally don't understand how: all I see is that visible features are increasingly harder to distinguish as feature map size reduces from 1280x768 to 75x50.

This is the original image:

Conv1_1

Conv3_1

Conv5_1


texts: Using machine learning specifically for feature analysis, not predictions

texts:
I'm new to machine learning and have spent the last couple months having a blast using Sci-Kit Learn to try to understand the basics of building feature sets and predictive models.

Now I'm trying to use ML on a data set not to predict future values but to understand the importance and direction (positive or negative) of each feature.

My features (X) are boolean and integer values that describe a product. My target (y) is the sales of the product. I have ~15,000 observations with 16 features a piece.

With my limited ML knowledge to this point, I'm confident that I can predict (with some level of accuracy) a new y based on a new set of features X. However I'm struggling to coherently identify, report on and present the importance and direction of each feature that makes up X.

Thus far, I've taken a two-step approach:

Use a linear regression to observe coefficients
Use a random forest to observe feature importance

The code

First, I try to get the directional impact of each feature:

from sklearn import linear_model
linreg = linear_model.LinearRegression()
linreg.fit(X, y)
coef = linreg.coef_
...

Second, I try to get the importance of each feature:

from sklearn import ensemble
forest = ensemble.RandomForestRegressor()
forest.fit(X, y)
importance = forest.feature_importances_
...

Then I multiply the two derived values together for each feature and end up with some value that maybe perhaps could be the information I'm looking for!

I'd love to know if I'm on the right track with any of this. Is this a common use case for ML? Are there tools, ideas, packages I should focus on to help guide me?

Thank you very much.


texts: Is it ok to use photo collages as dataset instead of single images for training the object detection ssd model?

texts:
Is it normal/better to use photo collages (multiple photos in one image) as a dataset instead of single images for training the object detection SSD model?
I am using Tensorflow Object Detection API to make an SSD model to identify and detect different leave diseases. My dataset consists of 256x256 leaves (plant village dataset), I want my model to be trained for detecting multiple objects as well with different sizes (most of the leaves are almost as big as the image itself - which gives bad accuracy for small objects) so I thought combining those pictures into the collages to include several leaves in each picture and creating a dataset with the following way, for each label I will use single leaf images, 4 leaves images (2x2 collage) and 9 images (3x3 collage). Do you think that it will give better results for multiple object detection or Should stick with single images and it won't change anything? If it works what would you recommend to set for the image resize option? Should it be still low like 300x300 or it will be better to have 800x800 since having 9 images in some part?


texts: Is ROC and AUC the only criteria for choosing a model?

texts:
If no, what are the other criteria. Please elaborate.
What should be minimum value of AUC to select a model.


texts: How to implement a convolutional autoencoder?

texts:
I would like to implement a convolutional autoencoder in Tensorflow, but it is not clear how the decoder part should work.

Each layer of the encoding, is a convolutional layer with activation function and then a pooling layer.

But how will the decoding work? I know that I have to add padding in each layer, but what will be the reverse of the convolution? How will it reproduce the original data from much less variables and the padding?


texts: How do you compare outputs from different augmentations for consistency training?

texts:
I'm trying to figure out how papers using consistency train on unsupervised data, and I'm stuck on how outputs from different augmentations are compared. Since the augmentations are transformations, I would figure that another transformation would have to be done to the outputs in order to compare them, but I can't find (or recognize) any mention of this.
What method is used to compare the predictions from different augmentations of the same image?
One example of this would be UDA

Xie, Qizhe, et al. "Unsupervised data augmentation for consistency training." arXiv preprint arXiv:1904.12848 (2019).

Another would be FixMatch

Sohn, Kihyuk, et al. "Fixmatch: Simplifying semi-supervised learning with consistency and confidence." arXiv preprint arXiv:2001.07685 (2020).


texts: I am a noob in the field of ML. I have been trying to classify the iris dataset. I managed to do it with backpropagation, and with 3 neurons in the hidden layer. But the mean square error that I get when I rerun the entire code is not consistent. It varies between 0 - 8%. Is this expected, or am I doing something wrong here?


texts: Find inputs that give highest variance in output space

texts:
I have a relatively simple problem that I think should have a known solution, but that I can't figure out. Any help would be greatly appreciated.
Basically, I have a function $f : \mathbb{R}^d \rightarrow \mathbb{R}^p$ which has vector input of features and vector output of images. Given a training set of combinations of the input variables and their corresponding outputs $\{x_i, y_i\}_{i=1}^N$, I would like to find the input feature (i.e. element of the inputs) that leads to the highest variability in the outputs, i.e. highest pixel-wise variance. Broadly speaking, I'm hoping to quantify the relationship between each feature and its effect on the variance of the output, to get an idea of how "sensitive" that feature is.
Methods like PCA wouldn't work because I am working in a supervised setting. Whereas PCA tries to find the directions of maximum variance of an unsupervised dataset, I am trying, in a sense, to find the directions of maximum variance in the label (output) space as it relates to the input space. Does an answer to this exist? Thanks in advance.


texts: Is it possible to use Inception Model in GANs (DCGAN) using PyTorch(or any other library)?

texts:
MAIN ISSUE:
Is it possible to use Inception Model (e.g. v3) for DCGAN using PyTorch(any other library)?
I've tried to find info how it could be implemented but nothing has been found.
It was explained how it works here . It seems that it's possible to use it for discriminator if the output is known.
QUESTIONS:

Is it possible to somehow apply the model to the GAN? (in particular for DCGAN) [what should be the labelled outputs?] [how does it implement for generator?]?
Can be the size of the input images decreased somehow? (it's written that input size of the image should be 299x299). But simple training takes quite a lot of time at images of size 64x64 in GPU processor.

Any hints are welcome.


texts: Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

texts:
Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields.

Let's also say the PII could include a wide array of information categories like full names, dates of birth, places of residence, driver's license numbers, passports, family member names, etc.

Let's not forget to mention that thousands upon thousands of collectors were used to input this data too, which introduces an additional layer of complication (e.g. inconsistent reporting, nuanced terminology, abbreviations, spelling errors).

And finally, let's say that this data is meant to be released to the public so the risks of re-identification for nefarious purposes are very high.

Given that background, how feasible would it be to apply machine learning here? What important limitations/considerations should be taken during this process? Can machine learning address the problem of PII in this case? Why or why not?


texts: Can a GAN-like architecture be used for maximizing the value of a regression predictor?

texts:
I can't seem to convince myself why a GAN model similar to regGAN  couldn't be modified to maximize a regression predictor (see the image below).  By changing the loss function to the difference between the current predicted value and the maximum predicted value generated so far, wouldn't gradient decent converge such that the generator builds the inputs that will maximize the prediction in the Discriminator CNN?

In math terms, the loss calculation would look like:

  yhat = current prediction
  ymax = best prediction achieved yet
  Loss = ymax - yhat
  if Loss < 0 then Loss = 0; ymax = yhat
  Back-propagate the loss using SGD

If the current predicted value is higher than the maximum predicted so far, then the loss is 0 and the loss function is updated.  Essentially, we are changing the objective from generating inputs that look real to generating inputs that optimize the complex function encoded in the CNN.


texts: Can tanh be used as an output for a binary classifier?

texts:
I am creating a binary classifier in Keras and here;s the code

model = Sequential()
    model.add(Dense(30, input_dim=60, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I was wondering, if the dataset's range is changed from [0,1] to [-1,1], shouldn't I be able to use tanh as the activation of the output layer? If so, what are the advantages of one activation over the other?


texts: k-means and perceptron generalization ability

texts:
I've implemented simple Widrow-Hopf perceptron and k-means clustering and compare results against MNIST data set. I didn't expect great results because of linear nature of these algorithms.  WH perceptron ended up with ~70% accuracy and k-means with ~50% (I have unit tests and compare with pure random chance 1/10 I suppose they do work somehow correct).

Also I gave "a hint" to k-means by set up initial centroids around different digits, so it can converge faster.

I suppose that k-means showed this result because digits are pretty much similar (6 is like 8 and 8 is like 9). Then I assumed that generalizations ability of WH perceptron is better than k-means. Did anyone see any article/book about this topic? I want to understand results from a rigor mathematical perspective.


texts: Optimal input setup for character-level text classification RNN

texts:
I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two approaches I've thought of:

Provide the whole 500 characters (one per time step) to the RNN, and predict a binary class, $\{0,1\}$.
Provide shorter overlapping segments (e.g. 10 characters) and predict the next (e.g. 11th) character. Convert this to classification by taking the test input and calculate the joint probability of the observed characters based on predicted next-character distributions.

The first approach seems sub-optimal as I don't believe that the 1st character is going to have any effect on the prediction of the 500th character. The second approach gives me diminishingly small likelihoods when you calculate the joint probability.
I'm aiming for a more nuanced language model akin to n-gram frequency counting. I'm using simple RNNs for now but intend to swap to either LSTM or GRU.


texts: Visualizing items frequently purchased together

texts:
I have a dataset in following structure inserted in a CSV file:

Banana  Water   Rice
Rice    Water
Bread   Banana  Juice

Each row indicates a collection of items that were purchased together.  For example, the first row denotes that the items Banana, Water, and Rice were purchased together.

I want to create a visualization like the following:

This is basically a grid chart but I need some tool (maybe Python or R) that  can read the input structure and produce a chart like the above as output.


texts: How do I plot the predictions made by a LinearRegression model?

texts:
I have made a linear regression using sklearn. I am wondering if there is a convenient way where I can plot the prediction versus a specified variable?

Alternatively, is there any other good way of visualising a linear regression model considering it involves n-dimensions?

Thanks for any help.


texts:
Is it correct to say that the lower the p-value is the higher is the difference between the two means of the two groups in t-test?
For example, if I apply the t-test between two groups of measurements A and B and then to two groups B and C and I find that in the first case the p-value is lower than the second case, could one of the possible interpretations be that the difference between the means of group A and B is greater than the difference between the means of group C and D?


texts: How to build a Caffe dataset for object detection with ImageNet data

texts: I am new into Caffe and I have some trouble in understanding how should I create a dataset for object detection. I have downloaded some images and bounding boxes from ImageNet. The bounding boxes are in Pascal VOC format (xml files). As I understood, this is not the only format for bounding boxes. However, I want to build a dataset that can be fed to a Caffe network. Again, I know that there are several suitable formats. For now, I am not interested in performance. What is the easiest method to build the dataset?


texts: What does it mean by “t-SNE retains the structure of the data”?

texts:
I was learning about t-SNE when I was told that t-SNE retains the structure of the data in the embeddings.

What exactly does this mean ?

How does the algorithm achieve this ?

So far I have understood this ->

t-SNE is an unsupervised learning algorithm that is used for dimensionality reduction and visualization of high dimentioanal data. The algorithm works by measuring the similarity between one point and all the other points using the t-curve. The width of the curve is dependant on the density of the cluster the point belongs to. t-SNE retains the structure of the initial data.

My question is what does it mean by retaining the structure of the data? Shouldn't there be some loss in the structure of the data seeing as it is transformed into a lower dimensional space ? Also what does "strucutre of the data mean"? Please ask for any further details that are required.


texts: What is the first tool to learn start your your data science projects?


texts:
I have an idea but I am not certain that it can be modeled in a DL architecture.

Let's say we have images of different qualities based on color patterns and their assessment as labels in a range from 0-1.
E.g. Image 1 has 0.25 quality, Image 2 has 0.5 quality and so on.

Could this be implemented in a standard Resnet50 architecture with 1 output and sigmoid?
Is there anything in literature that you could point me up to? I wasn't able to find anything, maybe I am searching it wrong.

EDIT: I have found this https://github.com/idealo/image-quality-assessment, but the implementation is different that I suggest. I know that this can work if I add five different quality classes.

I want to know if I can train this with 1 class output with a rank, in order to let the model understand how those qualities are linked. For instance quality 2 is the next best quality than quality 1.


texts: Linear Regression: Why use global basis functions instead of local basis functions

texts:
I'm looking through an online course about machine learning and the first big topic is finding a model that approximates our data with linear regression.
The model itself is linear function and we look for the coefficients of that linear function.
We introduce non-linearities that the data might have into our linear model by adding feature crosses.

For example, lets say we have two-dimensional features ($x_1, x_2$) and label them as $1$ if they are in the 2nd or 4th quadrant, 0 otherwise. A linear function $y(x_1, x_2) = b +m_1x_1+m_2x_2$ cannot approximate this. If we add $x_3 = x_1x_2$, we can find a model that represents that.

However, this is obviously a very artificial problem and not at representative to solve against any kind of data. In the end, we are trying to find a function $y$ that fits some data. This reminds me of solving differential equations numerically. And there we build the function $y$ as a sum of local basis functions $y(x) = \sum_i m_i\phi_i(x)$.

Why would we use global basis functions instead of local basis functions to introduce non-linearity into our linear machine learning model? Why don't we discretize our feature space and optimize on a linear function space?


texts: Feature encoding for multiple JSON objects

texts:
I have a dataset, where a particular feature is a collection of many JSON objects for a single feature.

Timestamp    Observations
1            {"name":"bob", "place":"TX"},{"name":"ann", "place":"NY"},{"name":"jack", "place":"MA"},{"name":"jill", "place":"CA"}
2            {"name":"bob", "place":"TX"},{"name":"ann", "place":"NY"},{"name":"jack", "place":"MA"},{"name":"jill", "place":"CA"}, {"name":"darth", "place":"XX"}

How can I encode the Observations feature for each tuple, so that I can use it to do some sort of anomaly detection? (Note that for the second tuple, there is an extra JSON object)


texts:
I have the objects pool with histories of their states, where each transition from one state to another takes some time, ex:
$$
object\ 1: A \overset{1s}{\rightarrow} B \overset{2s}{\rightarrow} C
\\
object\ 2: D \overset{3s}{\rightarrow} E \overset{4s}{\rightarrow} F
$$
The pool consists of many objects with right histories, where sequence of states and transition duration follows some unknown rules.
Also the pool consists a small amount of object with wrong history. But which objects are right or wrong is unknown.

It needs to found mistakes in the pool. Ex, "the transition from X to Y is not allowed" or "the transition is too long for such type".

What machine learning models are more suitable for this problem?


texts: Should we identify outliers from population prior to taking sample?

texts:
I am revising undergrad statistics course via this course, where i am learning techniques to pull out sample from population.

While ensuring that sample is a decent representative of the population, I am left with a question.

Should we care about identification and rectification of outliers prior to taking sample from population ?

Here is my work from where i came up with this question


texts: Is it acceptable for a forecasting model to predict moving averaged version of the data?

texts:
I'm working on a project in which I'm developing a precipitation forecasting model.

When I try to predict the original data, the model (LSTM neural network) is not able to predict the peaks.

This is the predicted values for the original data:

 So, I decided to perform

1) A differencing method in which I simply subtract values of the past from current values.

2) A moving average method on the resultant data from step 1.

Then I try to predict this processed version of the data.

Below is the predicted values for the smoothed data:

1) Is this kind of forecasting models acceptable among academicians?

2) should I try just to predict the original data?

3) What probable usage can these types of predictive models (which forecasts smoothed version of the data) have?

4) Any suggestion on how to deal with this situation?


texts: Given time series data, how to model the frequency of someone changes his job?

texts:
I am given a time series data vector (ordered by months and years),which contains only 0s and 1s. 1 s represent a person changes his job at a particular a month.

Questions: What model can i use to determine model how frequently this person change his job ? In addition, this model should be able to predict the probability of this person changing his in the next 6 months.

A poisson process ? (I have studied poisson process before however I have no idea when and how to apply it). Any assumptions that data need to meet before applying the poisson process ?

Would love to gather more information on how to model something like this. Thanks


texts: How to add Error Bar to Matplotlib line plot?

texts:
I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this plot which shall show the standard deviation. I have looked up to different answers but in most of them they had defined x and y explicitly, but here I calculate the plot directly from the dataframe. How to add error bar to this plot?
Dataframe df
UserId     |   date                 |-7|-6|-5|-4|-3|-2|-1|0 |1 |2 |3 |4 |5 |6 |7
     1      2009-10-17 17:38:32.590 |0 |0 |0 |0 |0 |0 |1 |0 |1 |0 |0 |0 |0 |0 |0
     2      2009-10-19 00:37:23.067 |0 |0 |0 |0 |0 |1 |1 |0 |1 |0 |0 |0 |0 |0 |0
     3      2009-10-20 08:37:14.143 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0
     4      2009-10-21 18:07:51.247 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0
     5      2009-10-22 21:25:24.483 |0 |0 |0 |0 |0 |0 |1 |0 |0 |0 |0 |0 |0 |0 |0

Code
badges = ["A", "B", "C"]

for badge in badges:
  res.iloc[:,2:].mean().plot(kind='line', label = badge)

Output


texts: Does experience with Keras count as experience with Tensorflow?

texts:
Many Machine Learning job postings I've seen request experience with Tensorflow. If I have experience with Tensorflow, but only through building neural networks using the Keras API. Does that count?
I have yet to see a tutorial or any code anywhere that uses Tensorflow without the Keras API, so I don't see how one learns Tensorflow strictly. But then why don't employers just request experience with the Keras API?


texts: Problem about machine learning regression

texts:
I have to predict the weight of fishes in a database. With "mae" i have to improve getting less than 10..but I dont know what kind of optimizer i have to use for improve the "mae"..
159 fishes in database
7 attributes (species, Lenght1, Lenght 2, Lenght 3, Height, Width.
Training set -- 143 instances
Validation set --- 16 instances
Full code:
from google.colab import drive
drive.mount('/content/drive')

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = pd.read_csv('/content/drive/My Drive/G2/Fish.csv', header=0)

from sklearn.model_selection import train_test_split
y=df['Weight']
x=df.drop(columns=['Weight'])

#Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train = sc.fit_transform(x_train)
x_test = sc.transform(x_test)

from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.callbacks import ModelCheckpoint

def build_model():
    model = models.Sequential()
    model.add(layers.Dense(64, activation='relu',
                           input_shape=(x_train.shape[1],)))
    model.add(layers.Dense(64, activation='relu'))
    model.add(layers.Dense(1))
    model.compile(optimizer='rmsprop', loss='mse', metrics=['mae']) #MSE = Mean Square Error / MAE = Mean Absolute Error
    return model

from keras import optimizers

model.compile(optimizer='RMSprop',
              loss='mse',
              metrics=['mae'])

es = EarlyStopping(monitor= 'mae', mode='min', patience = 100, verbose=1)
mc = ModelCheckpoint('/content/model_regression.h5', monitor='mae', mode='min', verbose=1, save_best_only=True)
history = model.fit(x_train,
                    y_train,
                    epochs=150,
                    batch_size=10,
                    callbacks = [es, mc])

Final Result:
Epoch 00150: mae did not improve from 29.04137  ------> The number I get is 29, i have to improve this in less than 10
Any idea? Thanks.


texts: LSTM future steps prediction with shifted y_train relatively to X_train

texts:
I'm trying to predict simple one feature time series data with shifted train data. The source looks like this:
   DATE              PRICE
0  1987-05-20        18.63
1  1987-05-21        18.45
2  1987-05-22        18.55
3  1987-05-25        18.60
4  1987-05-26        18.63

Actual code with data link download gisted here: gist
So the main problem is that it actually can't predict next steps. Roughly speaking: y_train "shifted" relatively to X_train by timesteps defined in parameters. So we getting for X_train and y_train something like this:
timesteps = 5
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
# After manipulations which you can find in gist we getting this:
X_train =
    [[ 1  2  3  4  5]
     [ 2  3  4  5  6]
     [ 3  4  5  6  7]
     [ 4  5  6  7  8]
     [ 5  6  7  8  9]
     [ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]]
y_train =
    [[ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]
     [11 12 13 14 15]
     [12 13 14 15 16]
     [13 14 15 16 17]
     [14 15 16 17 18]
     [15 16 17 18 19]]

So it is fair to assume that after training LSTM model with X_train (as input) and y_train (as output) we getting model which able to forecast n timesteps ahead. BUT I encountered a problem that trained model not predicting anything - only "duplicates" X_test data. For the convenience
I rebuild X_test data and plot it with y_test data which returns from model.predict():

So this is result which also contains 'Dataset prices' (pure data from dataset[upper_train + timesteps:]) for clarity.
I can not find where I made a mistake (or maybe this approach is bad?) so I will be grateful for any help!


texts: Ordering a material science dataset (properties names, properties scalars, formulas)

texts:
I'm dealing with a materials science dataset and I'm in the following situation,
I have data organized like this:
Chemical_ Formula     Property_name            Property_Scalar

    He                Electrical conduc.          1
    NO_2              Resistance                  50
    CuO3              Hardness
    ...               ...                        ...
    CuO3              Fluorescence                300
    He                Toxicity                    39
    NO2               Hardness                    80
    ...               ...                         ...

As you can understand it is really messy because the same chemical formula appears more than once through the entire dataset, but referred to a different property that is considered. My question is, how can I easily maybe split the dataset in smaller ones, fitting every formula with its descriptors in ORDER? I really need help on this... thank you.  ( I used fiction names and values, just to explain my problem.)
I'm on Jupyter Notebook and I'm using Pandas.
I'm editing my question trying to be more clear:
My goal would be to plot some histograms of (for example) n°materials vs conductivity at different temperatures (100K, 200K, 300K). So I need to have both conductivity and temperature for each material to be clearly comparable. For example, I guess that a more convenient thing to obtain would be:
Chemical formula     Conductivity      Temperature

      He                 5                  10K
      NO_2               7                  59K
      CuO_3              10                 300K
      ...                ...                ...
      He                 14                 100K
      NO_2               5                  70K
      ...                ...                ...


texts: Doing q individual train/test splits for q possible labels in a multilabel problem?

texts:
Let's say that I have a multilabel problem, where each sample can be of class A, B, C, or any combination of these.

Because of high imbalance, I've found that if I tackle the problem as 3 separate, binary problems (is it A/B/C?) I actually get a decent performance, compared to multilabel-adapted algorithms.

But I'm wondering whether it makes a difference to

A) Make a global train/test split, and then deal with each class individually for both train and test. Each classifier is trained on the same data, but with different goals. This is how e.g. skmultilearn seems to prefer it.

B) Treating the three problems entirely separately, i.e. split separately, preprocess separately, validate separately. The three classifiers never share any data, so there shouldn't be any leaking of knowledge. The final performance is also evaluated on different parts of the data, but since the classifiers are entirely independent, this shouldn't matter.

The second option should, according to my intuition, give the best representation for each binary problem, with the only downside being potentially 3x the processing. However, as my problem concerns only a few hundred samples, computing time is negligible.


texts: Why not use constant instead of permutation for model agnostic predictor importance?

texts:
I want to determine predictor importance. Ideal is to re-train same model on same dataset missing each variable in turn. This is too time consuming. The recommendation I have seen everywhere is to "remove" the column by converting it into noise by replacing it with its permutation. Why is it not better to replace the variable with a constant, thus "muting" the signal?

I ran an experiment on my own natural dataset with highly cross-correlated variables removed. Variable importance was computed using the constants 0, mean median and values common in variable. I used all of caret's regression models. Loss function is Pearson's correlation between supervised target variable and model prediction with one "muted" variable using different constants or permutation. Data with no error (variable did not matter one bit) was removed. All center constants produce less of a drop in correlation than permutation produced about 900/1400 of the time. 531/1421 constants did not make same error therefore interaction and conditionality. At least all decision trees would do this. If variable is part of polynomial as an additive term, replacing it with any constant would produce same correlation.

From ESLII, in a random forest," The randomization effectively voids the effect of a variable, much like setting a coefficient to zero in a linear model (Exercise 15.7)." Or the variable to it's mean.

Take all available values in a variable and use them as the muting constant. Weigh by constant's occurrence. The mean of that is what permuting converges to as more iterations are run. Right?

I can not think of what to try next to understand this problem.


texts: Is machine learning suitable for detecting screen elements?

texts:
I want to test a few devices as pure black box tests. That would mean some kind of robotics to press buttons and the touch screen and detecting screen elements like buttons in varying light conditions with a video camera. I can handle the mechanics but I am not sure how to got about screen detection.

Would machine learning be suitable for such a task? I imagine training the system under different lighting conditions.


texts: Which Random Forest hyperparameters to tune with Grid Search and which are the best initial hyperparameters values?

texts:
I want to use Grid Search for finding optimal hyperpameters for Random Forest. My questions are:

Which Random Forest hyperparameters are considered important for tuning?
Which initial Random Forest hyperparameters values for Grid Search do you recommend?


texts:
I have a dataset with 5K records for binary classification problem.

My features are min_blood_pressure, max_blood_pressure, min_heart_rate, max_heart_rate etc. Similarly, I have more than 15 measurements and each of them have min and max columns amounting to 30 variables.

When I ran correlation on the data, I was able to see that these input features are highly correlated. I mean min_blood_pressure is highly correlated (>80%) to max_blood_pressure. Each measurement with its min and max feature is highly correlated. Though their individual correlation to target variable is less.

So in this case, which one should I drop or how should I handle this scenario?

I guess there is min and max variables for a reason. How would you do in a situation like this?

Should we find the average of all the measurements and create a new feature?

Can anyone help me with this?


texts: How can I perform weighted addition of three layers in keras?

texts: I would like to perform the weighted addition of three outputs from different Keras layers such that the weights are trainable. How can I achieve this? I am using tensorflow 2.0 as backend for Keras.


texts: Unsupervised Function Optimization using Input and Output for Loss Function?

texts:
I have some vectors {$\mathbf{X_1 ... X_n}$} and they are all of dimension 1 x N. Vectors {$\mathbf{X_1' ... X_n'}$} are also 1 x N and are related to {$\mathbf{X_1 ... X_n}$}, but the relation cannot be modeled by a function.

I want to train a neural network such that for each $\mathbf{X_i}$ I input, it gives me a $\mathbf{Y_i}$ where the loss function to optimize is $\mathbf{X_i' Y_i}$. The reason I do not use $\mathbf{X_i'}$ as the input is that I do not have access to them during testing. The constraint on $\mathbf{Y_i}$ is that the norm is 1.

I tried an implementation similar to this post here (https://stackoverflow.com/questions/46464549/keras-custom-loss-function-accessing-current-input-pattern) which is:

def custom_loss_wrapper(input_tensor):
    def custom_loss(y_true, y_pred):
        return keras.losses.mean_squared_error(y_true, y_pred) + f(input_tensor)
    return custom_loss

However, I found that adding f(input_tensor) only changes the calculated loss, and not the back-propagation itself. The neural network produces same output $\mathbf{Y'}$ with or without the f(input_tensor).

I also tried direct optimization without a neural network, using TensorFlow optimizer.minimize() or similar strategies, and the results are not very good.

Any ideas how I may build this network?


texts: How to extract assignment from natural language text?

texts:
I'm a bit new to NLP/IE.
I'm looking for the task within NLP/IE that would be concerned with extracting a value that has been assigned.

For instance, given the text "The value is 45.1hz" or "The color is blue", I would like to be able to extract the fact that 45.1hz has been assigned to the value, or that the color has been set as blue.

I have tried to find the right name for this task - It seems somewhat related to relation extraction, however it doesn't seem to have two or more entities, and the relation unclear to me.

It may also be related to triplet extraction, but once again I'm a bit loss on what the values would be, i.e. (color, be, blue) ?

I'm looking for the name for this type of task, where you extract the value of an assignment, for a specific pre-defined value.


texts:
I work in an analytical role at a a large financial services firm. We do a ton of daily reporting over metrics that rarely change in a meaningful way from day to day. From this daily reporting, our management is required to extract what was important yesterday and what important trends have developed / are developing over time.

I want to change this to a model of daily exception reporting and weekly trend reporting.

Features might include:

User report consolidation (so there's only one daily email)
report ordering based upon level of variance from past performance (see the most important stuff first)
HTML email support (with my audience, pretty counts)
Web interface to allow preference changes, including LDAP support (make administration easier)
Unsubscribe feature at the report level

Here's what I'd like to know:

What are the practical problems I might run into?
What is the best way to display the new reports?
How should  I define an "exception"? How can I know if my definition is a good one?
I assume I'd be using a mix of Python, SQL, and powershell. Anything else I should consider, e.g. R? What are some good resources?


texts:
Imagine I have a collection of data, let's say the travel time for a road segment.
On this collection I want to calculate the mean and the standard deviation. Nothing hard so far.

Now imagine that instead of having my collection of values for one road segment, I have multiple collections of values that correspond to the multiple sub segments that compose the road segment.

For each of these collections, I know the average and the standard deviation.
From that, I want to aggregate these multiple average and standard deviation in order to get the average and standard deviation for the whole road segment.

For example, let's suppose I have the following dataset :

           subSegmentA , subSegmentB , subSegmentC , subSegmentD
values              20            45            25            70
                    30            55            10            60
                    10            10            10            80
                    15            50            30            75
                    15            40            15            75
                    20            40            20            80
                    30            45            20            65
                    10            40            25            70

average          18.75        40.625        19.375        71.875
stddev      7.90569415   13.47948176   7.288689869   7.039429766

expected_global_average : 150.625
expected_global_stddev  : 18.40758772

For the average there is no problem, a simple sum do the job, but I have trouble with the global_stddev.

I tried multiple solutions from here, without success.

Edit :
After further research, it seems mathematicaly impossible to calculate the standard deviation of a set based only from the standard deviation and average of subsets.

So I am trying to calculate a new metric, that would approximate this global standard deviation.

To do so, I can use in addition to the avg/stddev per subsegments, the length ratio of the subsegment to the road.


texts:
What are x variable and y variable in word2vec model if it is supervised learning.
In both the flavours- CBOW and skip-gram model.

Though some blogs have explained it as unsupervised learning.

Thanks


texts: how to predict an image using saved model

texts:
from keras import optimizers
from keras.models import load_model
from keras.preprocessing import image
import numpy as np
import cv2
import scipy.misc
from keras.wrappers.scikit_learn import KerasClassifier
# dimensions of our images
img_width, img_height = 313, 220
# load the model we saved
model = load_model('model.h5')
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy','mse'])
test_image= image.load_img('/Images/1.jpg',target_size = (img_width, img_height))
#x= scipy.misc.imread('/Images/1.jpg').shape
test_image = image.img_to_array(test_image)
print test_image
test_image = np.expand_dims(test_image, axis = 0)
test_image = test_image.reshape(img_width, img_height*3)
result = model.predict(test_image)
print result

When I ran the above code I get this error:

  ValueError: Error when checking : expected dense_1_input to have shape (36,) but got array with shape (660,)


texts: How can one generate future forecasts from probabilistic events?

texts:
I have an event "whether an item sold will be returned or not" which I can predict with a certain probability based on information gathered at the time that the purchase occurs (product features, customer information, time and place, etc). So:

P(Return | transaction information) = x% for a specific unit sold

I also have a historical time series of total units sold for that item, and a future forecast of sales of that item over the next few weeks. Assuming I gave the relevant transaction data for each historical sale that occurred, is there away to generate a future forecast from the return probability, so that I can state with some confidence that I will get 15% total returns on the item next week, 10% the week after etc?


texts: Retrieving image masks from PASCAL and COCO segmentation annotations

texts:
How do I retrieve image masks for instance segmentation data which has annotations saved in the COCO and PASCAL annotation formats with python?
The format is very confusing for instance segmentation data, I understand the bounding box formats easy.


texts: VAE KL-divergence with non-standard mean

texts:
I know I can make a VAE do generation with a mean of 0 and std-dev of 1.
I tested it with the following loss function:
def loss(self, data, reconst, mu, sig):
    rl = self.reconLoss(reconst, data)
    #dl = self.divergenceLoss(mu, sig)
    std = torch.exp(0.5 * sig)
    compMeans = torch.full(std.size(), 0.0)
    compStd = torch.full(std.size(), 1.0)
    dl = kld(mu, std, compMeans, compStd)
    totalLoss = self.rw * rl + self.dw * dl
    return (totalLoss, rl, dl)

def kld(mu1, std1, mu2, std2):
    p = torch.distributions.Normal(mu1, std1)
    q = torch.distributions.Normal(mu2, std2)
    return torch.distributions.kl_divergence(p, q).mean()

In this case, mu and sig are from the latent vector, and reconLoss is MSE. This works well, and I am able to generate MNIST digits by feeding in noise from a standard normal distribution.
However, I'd now like to concentrate the distribution at a normal distribution with std-dev of 1 and mean of 10. I tried changing it like this:
compMeans = torch.full(std.size(), 10.0)

I did the same change in reparameterization and generation functions. But what worked for the standard normal distribution is not working for the mean = 10 normal one. Reconstruction still works fine but generation does not, only producing strange shapes. Oddly, the divergence loss is actually going down too, and reaching a similar level to what it reached with standard normal.
Does anyone know why this isn't working? Is there something about KL that does not work with non-standard distributions?
Other things I've tried:

Generating from 0,1 after training on 10,1: failed
Generating on -10,1 after training on 10,1: failed
Custom version of KL divergence: worked on 0,1. failed on 10,1
Using sigma directly instead of std = torch.exp(0.5 * sig): failed

Thanks.


texts: I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/deep learning when should be the case that it be must to use Apache Spark?


texts: Batch Normalization and Input Normalization in CNN

texts:
I build my CNN on Keras, normally in the ImageDataGenerator I saw the rescale = 1. / 255 used to normalize input data (pixel value) from [0-255] to [0-1]. Then I read about Batch Normalization Layer, I wonder if they are mutually-exclusive or can they be used together in the same network?

I have tried to adapted the BN Layer to my network (BN after every activation Layer) but the loss fluctuates more than without using BN.


texts: Why does my loss value start at approximately -10,000 and my accuracy not improve?

texts:
I'm developing a multi-label classifier using the Keras library, but I am stuck with a relatively low accuracy of about 2% and my loss value per epoch is around -10,000 with little change between epoch. I'm relatively new to Keras but need to use it for my university work, and I am wondering what is going wrong.

My model currently looks as follows:

model = Sequential()
model.add(Dense(1024, activation='relu', input_shape=X.shape[1:]))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(3, activation='sigmoid'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])

model.fit(X, y, batch_size=20, epochs=100)

In this case, the input_shape[1:] is 4096, a flattened feature array and the multi-label I'm trying to classify has a first value that ranges from 1 - 163, the second value ranges from 1 - 2000, and the third value ranges from 2007 - 2014.

The issue that I am having is while running my model I get a result that does not change during each epoch as is shown below:

I don't know what I'm doing wrong, I'll include how the features and the labels look below in respective order:

[['0.0' ' 0.0' ' 0.0' ..., ' 5.7333' ' 0.0' ' 2.58643']
 ['0.0' ' 3.12623' ' 0.0' ..., ' 0.0' ' 0.0' ' 2.93147']
 ['0.0' ' 0.0' ' 0.0' ..., ' 7.49419' ' 0.0' ' 1.55746']
 ['0.0' ' 0.0' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.3666']
 ['0.0' ' 4.67996' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.0']]

[['78' '1' '2010']
 ['78' '1' '2010']
 ['78' '1' '2014']
 ['78' '1' '2013']
 ['78' '1' '2012']]

I'm at a loss of what to try and as I have mentioned, I'm new to Keras and deep learning so any help will be greatly appreciated.


texts: I want to use a neural network for a control block in MATLAB, but without Simulink

texts:
I have a physical model that produces an error value after running, which I want to minimize by a Neural Network. I can not find a MATLAB function however that lets me run the program and use the RMSE for training of a network.

How can I use my own model in a loop, combined with a network, so that the error is minimised?


texts: Why is Regularization after PCA or Factor Analysis a bad idea?

texts: I have done Factor Analysis on my data and applied various machine learning models on it. I particularly find it giving high MSE value for Ridge and Lasso Regression compared to other models. I want to know the reason why this happens.


texts: Finding the most important questions from a questionary-> results

texts:
Lets assume that I have 5000 articles and I create the TF-IDF of these articles.

Now I ask som people to answer 30 questions and I create the TF-IDF of these answers from the IDF of the articles and use the cosinus similarity to find the 20 closest articles per person with a weight of 1 for each answer. People can mark these 20 articles as relevant to her/him or not relevant(0 or 1).

So, which approach would you follow to see which questions are the most important in their choices? In my mind it seems like feature selection problem. Or not? Any recommendation?

Generally, what I want to do is reduce the number of questions from 30 -> 5 and I have to find which one are the most important questions.

Edit

As @null_9 mentioned in a comment, I am not sure which is the best way of providing the most relevant articles: 1) Summing up the similarity scores from all the answers and give the highest, or 2) find N closest articles per each question and sample 20 of them and provide to the user.


texts: How to deal with unbalanced class in biological datasets?

texts:
When dealing with unbalanced class, which is better,

oversampling/undersampling of the classes or
randomly selecting equal number of positive samples and negative samples from the training dataset and combining as training samples, transforming the imbalanced classification problem and replacing by multiple balanced data classification problems?

Does one of them have an advantage over the other? If so, which one? I am asking for a generalized point of view. If you had an unbalanced dataset, which option would you choose, 1 or 2?


texts: What is the difference between cache() vs prefetch() in tensorflow?

texts:
I have gone through the TensorFlow documentation.
What is the difference between cache() vs prefetch() in TensorFlow?
When should I use the cache() function and when should I use the prefetch() function?


texts: Input for individual perceptron in input layer in MLP

texts:
For better understand of neural networks I started implementation of Multi Layer Perceptron. For now I'm implemented single Perceptron that resolve XOR problem. From this point I want start build MLP but I'm not sure if I correctly
 understand MLP structure. Assume I have data instance with 3 attributes and 2 classes and 5 input perceptrons. Should I give every perceptron same input and every output of perceptron should go to every hidden layer perceptron's input? I attach neural net pic.


texts:
This is referencing Prof. Andrew Ng's course on machine learning.

In the part that details implementing an SVM with the Gaussian kernel, we are supposed to use all the training examples as our landmarks/support vectors to calculate the Gaussian kernel.

Specifically, for each training example, we will get a $m+1$ dimensional vector (including a bias unit) when we calculate the Gaussian kernel for each training example with every other training example (including itself). So, will doing this with every example not result in a $m+1 \times m+1$ matrix instead of a $m+1$ dimensional feature vector?

According to my understanding, we need $m+1$ features for us to be able to write the cost function. However, here, won't we end up with $m+1 \times m+1$ features instead?


texts: Best algorithm/model to establish relevance between events utilizing mixed data type (Tags, Time, x_coordinate, y_coordinate)?

texts:
I'm building a relevance ranking system for incidents occurrence and prevention. My goal is to use four attributes to establish relevance: tag (About 500 tags), x_coordinate, y_coordinate and time. The final state that I would like to achieve is: to select an object and have a relevance list from the most relevant objects, that correlate to the one that I've selected, to the least relevant ones.
Ex.: I have a list of events such as event A that represents a fork-lift accident in x_c=5, y_c=7, 22:10:59, tag["Crash"]. event B (fork lift speeding at x_c = 3, y_c = 5, 22:10:11, tag["Speeding"]). Event C (overheating on fork-lift 22:01:44, x_c=-4, y_c=-1, tag["Mechanical Issue:Overheating"]). Now I select event A and then I get a list of the most relevant events related to event A, usually when you crash the fork-lift is because of high-speed, and they tend to be spatially and timely near each other. Therefore, by selecting event A in my list of relevant events I get event B before event C in my list. The issue for me is that by encoding the tags I don't know if I'm keeping them relevant for my model, and they are suppose to be very relevant...
The two different approaches that I used to achieve this goal were:
Multi-linear regression to create a correlation function between the objects. This works to a certain extent, the problem is to encode the tags in a way that doesn't negatively influence the correlation.
Unsupervised learning, utilizing clustering and utilize the inertia between points to establish the relevance between the objects: low inertia=high relevance. Same issue here with the encoding, I must encode both labels and time-stamps (HH:mm:ss).
Please share your thoughts and let me know if there is any better approach or more optimal way of executing the approaches that I'm already using (I've wrote the scrips in python, so it's easy and fast to make changes before implementing the final version in a different platform).


texts: Using TF-IDF with other features in scikit-learn

texts: What is the best/correct way to combine text analysis with other features? For example, I have a dataset with some text but also other features/categories. scikit-learn's TF-IDF vectorizer transforms text data into sparse matrices. I can use these sparse matrices directly with a Naive Bayes classifier for example. But what's the way to also take into account the other features? Should I de-sparsify the tf-idf representation of the text and combine the features and the text into one DataFrame? Or can I keep the sparse matrix as a separate column for example? What's the correct way to do this?


texts: What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

texts:
I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works linearly).
I have tried this in Google Colab by selecting GPU settings but still it is very slow. I also try "swifter.apply" but still it is not as efficient.
Is there any way to do this?


texts: Nested cross-validation and selecting the best regression model - is this the right SKLearn process?

texts:
If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best hyperparameters, and the outter loop (cross_val_score) evaluates the hyperparameter tuning algorithm. I then choose which tuning/model combo from the outer loop that minimizes mse (I'm looking at regression classifier) for my final model test.

I've read the questions/answers on nested-cross-validation, but haven't seen an example of a full pipeline that utilizes this. So, does my code below (please ignore the actual hyperparameter ranges - this is just for example) and thought process make sense?

from sklearn.cross_validation import cross_val_score, train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
from sklearn.datasets import make_regression

# create some regression data
X, y = make_regression(n_samples=1000, n_features=10)
params = [{'C':[0.01,0.05,0.1,1]},{'n_estimators':[10,100,1000]}]

# setup models, variables
mean_score = []
models = [SVR(), RandomForestRegressor()]

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)

# estimate performance of hyperparameter tuning and model algorithm pipeline
for idx, model in enumerate(models):
    clf = GridSearchCV(model, params[idx], scoring='mean_squared_error')

    # this performs a nested CV in SKLearn
    score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')

    # get the mean MSE across each fold
    mean_score.append(np.mean(score))
    print('Model:', model, 'MSE:', mean_score[-1])

# estimate generalization performance of the best model selection technique
best_idx = mean_score.index(max(mean_score)) # because SKLearn flips MSE signs, max works OK here
best_model = models[best_idx]

clf_final = GridSearchCV(best_model, params[best_idx])
clf_final.fit(X_train, y_train)

y_pred = clf_final.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))

print('Final Model': best_model, 'Final model RMSE:', rmse)


texts: Is class discrimination is possible in Class Activation Mapping(CAM)?

texts:
I worked with Gradient Weighted class activation mappings(Grad-CAM) to understand and implement interpretability in Deep neural networks. I can also switch to a particular class(by selecting the correct neuron in last SoftMax layer for gradient calculation) in Grad-CAM.
I didn't find any way to incorporate any way to discriminate class in CAM because we are just applying the dot product to Convolutional layer and weights to produce single layer heatmap.
I want to know is there any possibility for class discrimination in Class Activation Mapping? I appreciate links any libraries or research papers in this area.


texts: Extracting the features from several Auto Encoders

texts: I am trying to extract the features of sparsed 3D pictures via a Convolutional Auto Encoder(CAE). Dou to the high computational costs I can not train the CAE over all the samples. I prepared couple of CAEs each trained on a small portion of the samples. And I used the extracted features for binary classification and I get high accuracy on the test data set. I wonder if there is a way that I can combine the extracted features by each CAE to improve the quality of the features.


texts:
I have a dataset with three inputs X1,X2,X3. I tried to predict next value of X1 using lstm .
I wrote the code for lstm and while trying model.predict I got an error cannot reshape array of size 4 into shape (4,3)

Here my code:

data= pd.DataFrame(data,columns=['X1','X2','X3','X4'])
data.columns = ['X1', 'X2', 'X3','X4']
pd.options.display.float_format = '{:,.0f}'.format
data = data.dropna ()
y=data['Y'].astype(int)
cols=['X1', 'X2', 'X3','X4']
x=data[cols].astype(int)

data=data.values
scaler_x = preprocessing.MinMaxScaler(feature_range =(0, 1))
x = np.array(x).reshape ((len(x),4 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(0, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)

train_end = 80
x_train=x[0: train_end ,]
x_test=x[train_end +1: ,]
y_train=y[0: train_end]
y_test=y[train_end +1:]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))

num_time_steps=3
model = Sequential()
model.add(LSTM(4, return_sequences=True,input_shape=(num_time_steps,x_train_n.shape[2])))  # returns a sequence of vectors of dimension 32
model.add(LSTM(8, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(8))  # return a single vector of dimension 32
model.add(Dense(1))

batchsize = 1
model.compile(loss="mean_squared_error",optimizer="adam")
history = model.fit(x_train,y_train, batch_size = batchsize, nb_epoch=30,validation_data=(x_test, y_test),shuffle =True)

model.reset_states()
pred1=model.predict(x_test)

pred1 = []
for s in x_train:
   pred1.append(model.predict(s[0].reshape(x_train.shape[2], num_time_steps)))
   pred1 = np.array(pred1)

Error :

MY CSV FILE

Can anyone help me to solve this problem?


texts: Spatial index for variable kernel nonparametric density

texts:
I'm trying to build a nonparametric density function for a fairly large dataset that can be evaluated efficently, and can be updated efficiently when new points are added. There will only ever be a maximum of 4 independent variables, but we can start off with 2. Lets use a gaussian kernel. Let the result be a probability density function, i.e. its volume will be 1.

In each evaluation, we can omit all points for which the evaluation point is outside a certain ellipsoid corresponding to the minimum gaussian value we care about. We can change this threshold for accuracy or performance, and the maximum number of points inside the threshold will depend on the chosen covariance matrix of the kernel. Then, we can evaluate the distribution approximately using the subset of points.

If we use a fixed kernel, then we can use the eigenvalues and eigenvectors we get from the covariance matrix to transform each point so that the threshold ellipsoid is a fixed circle. We can then shove all the transformed points into a spatial index, and efficiently find all points within the required radius of the evaluation point.

However, we would the kernel to be variable for two reasons. (1) to fit the data better, and (2) because adding or modifying points would require the fixed kernel to be updated, which would mean that the entire data set would need to be reindexed. With a variable kernel, we could make new/updated points only affect the closest points.

Specifically, is there a spatial index that can efficiently find ellipses surrounding a given point from a set of around 10 million ellipses of different shapes and sizes?

More generally though, does my approach look sound? I am open to answers like "give up and precalculate a grid of results". Answers much appreciated!


texts: How to pass common inputs to a merged model in Keras

texts:
I'm attempting to merge a VGG-16 and ResNet-50 through concatenation. I was successful in training and saving the merged model. Here is the code snippet:

from keras import applications
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras.layers import Concatenate
from keras.models import load_model
# Loading the training data
img_rows=300
img_cols=300
channel = 3
num_classes = 3
batch_size = 10
nb_epoch = 10
#load the first model #########################################################
base_model1 = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))
#get the model summary
base_model1.summary()
#addind the top layers
x1 = base_model1.output
x1 = GlobalAveragePooling2D()(x1) #analogous to flatten()
model1 = Model(inputs=base_model1.input, outputs=x1)
model1.summary()
#load the second model ########################################################
base_model2 = applications.ResNet50(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))
#get the model summary
base_model2.summary()
#addind the top layers
x2 = base_model2.output
x2 = GlobalAveragePooling2D()(x2) #analogous to flatten()
model2 = Model(inputs=base_model2.input, outputs=x2)
model2.summary()

'''merge the models'''
mergedOut = Concatenate()([model1.output,model2.output])
#add a new dense layer and softmax
out=Dense(2048, activation='relu')(mergedOut)
out = Dense(num_classes, activation='softmax', name='predictions')(out)
#create the new model with three branches and one dense layer
model = Model(inputs=[model1.input,model2.input], outputs=out)
model.summary()
###############################################################################
#training the model
hist=model.fit([X_train, X_train], Y_train, batch_size=batch_size, epochs=nb_epoch,
                     shuffle=True, verbose=1, validation_data=([X_valid, X_valid], Y_valid))

However, the inputs to both the models are the same. I would like to use this as a common input to the merged model to avoid passing X_train and X_valid twice during the model.fit. How can I do this?


texts: what approach to use for find best customer out of data?

texts:
I'm working on this project where the objective is to find certain good leads/customers from the existing customer dataset. I tried the RFM method for scoring but there is no data regarding money or any quantity.

dataset has:

customer_id

date (date when meeting is done)
next_action_date(if any next_meeting_scheduled that day)
status( done, missed, cancelled)

So based on only meetings, I need to score a lead or customer. So I started with normal control flow statement like,

if meeting was scheduled and missed- 5 points
if meeting was scheduled and and completed - 10points
if if meeting was scheduled and and cancelled -2points.

basically, lot of these cases are there by which I need to score a customer.
So how should I write a python code for this, I mean what will be the approach for this. I can get the data from an api for a customer. What I thought is to create features like,

days since last visit
no of meeting done succesfully
and more and add all these in other table.
My problem is which kind of code I have to write for this? Please explain in simple words if someone can. I'm new at this.


texts: Can my neural network learn conditional rules when classifying?

texts:
I'm concerned that I'm attempting the impossible with my neural network. This is the scenario:
I have a 2D square world. In it, I create five circles of random size and position. I then classify one of them as the correct answer, based on the following rules:

If any circle's radius is > THRESHOLD, I choose the largest circle
Otherwise, I choose the circle with the origin nearest the center

I send the inputs as serial coordinates, like this: [X0, Y0, RADIUS0, X1, Y1, RADIUS1, ...].
The output is a one-hot array, e.g. [0, 0, 1, 0, 0].
I've modeled this in TensorFlow without success. My best scoring result appears to always choose the largest circle, ignoring the else clause of the arbitrary rule.
Am I fundamentally misunderstanding the capabilities of neural networks? I've tried many (many) different configurations (layer counts, node counts, activation functions ... you name it). All of my networks have been feed-forward, so far.
Thanks in advance for any insight!

Here are some details of my network and data:

I have tried with up to 500k cases. I separate 10% for generalization checks after training, and train on the remaining 90% with a 50/50 validation split.
I've tried with the test data weighted 75% toward rule A, 50/50, and 75% toward ruleB.
I've tried 0-10 hidden layers, and neuron counts from 2 to 256 (each hidden layer gets the same number of neurons).
I change the number epochs as time allows, but generally it's 10-100. My longest runs have been several hours (with giant case numbers, and dropouts to prevent overfitting).
I've used batch sizes of 1-50.
I've tried learning rates of 0.0001 - 0.1.
I'm currently using ReLU activation, initializing bias to const(0.1) and kernel w/ heNormal. I have tried several other approaches for all three.
I standardize the inputs to center on zero w/ variance of one.
The loss function is categoricalCrossentropy.
The optimizer is Adam.


texts: Some questions about feature hashing in the context of document classification

texts:
I'm trying to understand feature hashing, specifically in the context of document classification.

I'm under the impression that it is useful because:

it allows us to easily deal with 'new' words/features/predictors that we haven't seen before
it is rather efficient because it allows us to exploit sparsity which is common in document/word data.

Some questions:

Say my word/feature set consists of 50,000 unique words and I use 16 hash bits $(2^{16}=65,536)$. In this setting, is it likely each word gets
mapped to one of the 65,536 indices?
How does this aid us in the situation where I build a model and try to make a prediction on a document that contains a word we haven't seen before?
Say my feature set is 100,000 words and I still use 16 hash bits. Does this effectively mean I have (on average?) $100,000-65,536 = 34,464$ words mapped randomly on to other words?

It seems I'm missing crucial point or idea here. Can anyone recommend some good introductory material?


texts: FFR and FAR calculating for multiclasss biometric face recognition system

texts:
I am implementing a face recognition system using facenet and svc Ml algorithm
i have like 20 classes or more and I'm getting 98% accuracy
im trying to calculate the FAR and FRR and the EER
I'm assuming that the threshold is the probability of predicting is that correct ?

this is the code for calculating the FP,FN,TP,TN

FP = matrics.sum(axis=0) - np.diag(matrics)
FN = matrics.sum(axis=1) - np.diag(matrics)
TP = np.diag(matrics)
TN = matrics.sum() - (FP + FN + TP)
TPR = TP/(TP+FN)
cm = ConfusionMatrix(testy, yhat_test)

from this matrix

how to calculate FAR FRR EER from these ?
i tried using the probability predictions for each FP,FN,TP,TN
like this

a =[]
for i in range(len(TP)):
       a.append(max(yhat_prob[TP[i]]))

print(a)

the output is :

0.6994577123013918, 0.6994577123013918, 0.4916665680599575,
0.6994577123013918, 0.6994577123013918, 0.6994577123013918,
0.30549183912100375, 0.30549183912100375, 0.30549183912100375,
0.30549183912100375, 0.30549183912100375, 0.32500348178885874,
0.30549183912100375, 0.4916665680599575, 0.30549183912100375,
0.30549183912100375, 0.30549183912100375, 0.6994577123013918,
0.30549183912100375]

and I don't know what to do next in case this is correct


texts: Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'?

texts: Tom White's 'Hadoop: the Definitive Guide' has become a popular guide to the entire Hadoop ecosystem and earned a reputation as providing both a broad survey, as well as covering individual aspects of Hadoop in decent depth. Has anyone thus far attempted to provide the Spark equivalent?


texts: Hash function for text documents that maps similar documents to the same value

texts:
I have a web site that process text documents (typically 10-100 pages) submitted by users.  Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar documents to map to the same hash value.  I essentially want to know whether a user is resubmitting a slightly changed document or a new document.

I don't store the documents, so I can only compare hash values and I can't compare the documents to each other.

I've done a lot of reading about MinHash and LSH, but these all seem to be based on having a corpus of a large number of documents and then finding similar documents within the corpus.  I think these don't work for me because I need to compute my hash vector on a single document at a time without knowing anything about other documents.

In some ways I feel like this should be an easy problem.  Something like computing a hash of a bag-of-words vector, but I'm struggling to figure out a good way to do this.

My comparison is based on text and not meaning so I don't need anything like word embeddings.


texts: pairwise_distances with Cosine and weighting

texts:
Is there a way to get a weight into the
pairwise_distances(X, metric='cosine')

Potentially using **kwrds?
from sklearn.metrics import pairwise_distances

In the scipy cosine distance it's possible to add in an array for weights, but that doesn't give a pairwise matrix.
a = np.array([9,8,7,5,2,9])
b = np.array([9,8,7,5,2,2])
w = np.array([1,1,1,1,1,1])

distance.cosine(a,b,w)

Where w is the weights.


texts: How to determine the abnormality of a specific variable by taking into account all the other variables in the data?

texts:
I have an issue of machine learning/anomaly detection. Indeed, I have a variable Y and several other variables X. The purpose is to quantify the degree of abnormality of the data on Y but I have to take into account the values on the other variables (the relationship between Y and X).
Normally, an anomaly detection algorithm would find anomalies but on the whole data (Y + X), but in my case I want to zoom in on Y because it is a very important variable. If I wanted to quantity the abnormality on all my variables (Y + X), Y would be lost in the middle of all the variables.
It is not something strange because when you apply a linear regression Y ~ X, you can calculate the Cook distance which is a kind of "abnormality score" and it took into account the relationship between Y and X.
I hope it is clear!


texts: Units of mean decrease accuracy on a variable importance plot obtained from a Random Forests classifier?

texts:
To know the importance variable in a Random Forest I used The mean decrease accuracy and mean decrease Gini.
I would like to understand what are the x-axis units of the mean decrease accuracy and mean decrease Gini on a variable importance plot obtained from a random forests classifier. For example the mean decrease accuracy and gini values range between 0,000 - 0,012 and 0-600 respectively.

Thanks


texts: Is it bad practice to use multi-class over multi-label classification?

texts: I have a multi-label classification problem-- millions of records that potentially could hold more than one label. I'm running into issues related to lack of research/examples online, and am unable to create a strong model. Would it be advised against to build a multi-class classification model, and take any label with a predicted probability over 0 to be considered a potential label? Even with problem transformations, like binary relevance and classifier chains, the accuracy is terribly low (~12%) as opposed to using Naive Bayes and getting ~57% accuracy.